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ABSTRACT . ^ , ^ ^ 

The purpose of this paper is to outline and ^xplain 
the evaluation system used by the Ehode Island Department of' 
Education to measure the eff ectivenes^s of compensatory education 
programs. An exposition of how relevant background, program and 
financial information is collected is presented* In addition, a 
delineation is made of the methods variables thaj^ distinguish 
successful from uns^uccessful programs « All. results are for fiscal 
year 197^4*75 projects* (Author) * ' 
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A Paper Presented at the Ameijican Educational Research Association Annual Meeting 

,San Franciscq,. California ^ ' ' . . 

o ' April 'f 9-23, 1976 ' 



INTRODUCTION , . 

The major purpose of this paper f - to outline and explain the data 
collection and Evaluation system used by the Rhode Island Department of 
Education to describe and .measure the effectiveness of compensatory 
education (Ti|:le I, ES?A aad State Section A) projects and-to point to' 
future directions that should be attempted. Results of most'' analyses 
are not presented here since the volume of iTiformation would be ixi- 
hibitlve, although reading gains achieved through the program are 
mentioned. 

BACKGRQUMD 

There have been several attempts to determine the effectiveness 
of compensatory education programs. at the national level. Studies on 
Head Start ptograms -indicated that t^he positive effects disappeared 
*by fbe'end of- first gxadev' Numerous national evaluations of Title. I, 
ESEA reading programs have reported non-significant ^results,- S^ome 
mentioned that program participants continued to decline, others 
mentioned that month for month gains were not achieved by participants 
and still others suggested ho rjelation between the es^ent of partici-* 
pation in compensatory education and achievement* For the most part, 
large scale evaluations have shawn discouragihg results* 

Those with greater faith an compensatory education rallied to its 
aid ^citing the methodological and statistical shortcomings of the 
studies. The list was lengthy, all aspects of these studies were 
scrutinized and, intone instance, even the expertise of the evaluator 




was questioned * The ilst of shortcomings can be reduced to these 
common problems: 

1) reporting pre-test and post-'test scores for different 

i ^ * \ 

children, ° * ' , 

2) mixing of test scores from different tests as a basis for 
obtaining overall'^v^erage gains, 

3) ' using or misusing grade equivalent ^scores to obtain average 

4) bsing tests without adequate floor properties, 

5) collecting data from unrepresentative samples, 

^ 6) selecting students on a test score and including that , ^ 

- . score as a pretest (regression toward the mean), 

7) clerical and scoring errors, ^ 

8) insufficient IcnowU^dge o§ the programs students are enroJUed 
in, ^ ^ . ^ . . t 

9) ' poor test administration, and 

1^ ' "* 

10) lack of adequate control or comparison groups. 

Annual ^tate evaluation reports of Titl^ X have typically shown 
more positive results than national evaluations, however, these documents 
also have been hindered by the problems listed above* The Rho5^ Island 
Department of Education has instituted a system of data collection and 
evaluation on a statet^lde basis tha^ eliminates many of the problems 

^^ 

that have plagued* similar efforts* 



'0 



ERIC 



THE INSTRUMENTS AMD COLLECTION TOffiS 

__ . , ( 

1* Pre-progjam information form — ^arly in SeptemberXlQcal system 

. ^ Title I coordinators indicate the naiaes of the reading teachers 

■ 

involved in each project; and the approximate number of students 
served by each teacher- * This7 information, is used to ijevelop 
packets of pre-test reporj;in^* instruments for each teacher* 

2* Pre-test reportmg^ instrument — Packets for reading teachers are 

distributed at a statewide meeting for coordinators in September* 

>i ■ " - * 

Foi^s are completed for each student participating in compensatory 
education reading programs. In addition to raw^ scores on prescribed 
tests* (see data pragaratdon secticfn), data oiv age, %ex^ ethni'c g^roup, 

prior Title I experience, type of school and g^r^de -retention^^are 

* . ** 

also collected for each child* 

* . * 4 

3, Post-test reporting instrument — Post-test information forms are 

distribute4 at a statewide Title I conference in May*_ Computer , 

forms generated from pretest information are completed by teachets 

and, in^addition to test scores^ data on .variables like abse:^teeism, 

^ activity, the child participated in, numfcer of weeks in program,'' etc* , 

are collected for each child who tc^k a pretest* 

ft * " . - 

4* Program Questionnaire for Teachers — In April, teachers dn Title 1/ 

Section 4 Reading programs are asked to complete a quf^stionnaire 

dealing with program variables Ixke minimum amount of time spent 

with any one child per week, sise of group serviced, type of materials 

used^ are checklists^ of reading progress used, etc* 

5 ■ ■ 



5, Parent Involvement Instrument — In May, a questionnaire is completed 
concerning the -duties, activities, and^functions of the parental' 
advisory committee as well as" about parental interaction with projeafr, 
operation* 

J ' ^ , ' , . ' ^ 

6* Combined Project and Program Information, Porm 74-75 — In June, 

project directors complete this instrument outlining expenditures 

by activity, children served by grade and activity, time of operation, 

full-time equivalents for personnel, nature and number o^f inservlce 
J' » ' . 
,^trainlng programs, etc, ^ > * ' . , 

. " ' % ■ ■ 

7- Individual project evaluations --^ Each project^is evrflliated, either 
* internally tOj-^the proj^^it or by an external agency^ ''f o prQvide 
information on the effectiveness of individual programs* 

TESTING INSTRUMENTS 

In an effort to circumvent the- prablems associated with combining^ 
tesfresults^'f rom different rfeadiifig instruments, a core group of local 
system administrators and teachers, evaluators and staff was developed 
for the "purpose of selecting, one Standardized. acblev.ment test -suitable -. 
to be used uniformly across the ^state for -compensatory education evalua- 
tion of reading at the state level*. 

The groups chose the following instruments and forms: ^ ' 
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TEST FORM 
GRADE LEVEL INSTRUMENT LEVEL PRE POST 

K California Tests of Basic Skills A S S 

1 California Tests of Basic Skills B S S 

2 California Achievement Tests 1 ^ A B 
3-4 California Achievement Tests 2 A B 
5-6 California Achievement Tests 3 A R 
7-9 , ^ California Achievement Tests ^ 4 ^ A B 
10-12 ^ ^ Cali^ojrpia Achieiiejoapt Tests 5 ^ /i ' "B 

In this way scores can be aggregated across the state for various grades* 

- PREPAfeATION- PROCEDURES 

All ^questionnaires and instruments tt^at are completed are thoroughly 
checked by a 'data. clerk priorto keypunching* The data clerk scrutinizes 
the returns fi5r -fe^^any blanks that have o'ccurre'd, and b) any glaring . 
errorjs 'in responses I'ike' fadli^jfe po rights-justify, etc- Any deldnquent 
retpjxi or inaccurate data is followed up. ■ i-. . . . 

After careful checking* the ^data is' sent tb keyp^unchers and then 
delivered to computer 'analyst^ * . ' 

ANALYSIS ^ ' ' - . . . 

Raw scores for every child in the program are submitted to the^ state 



Agency, <Raw scopes are converted to Achievement Development Scale Scores 

^ ^ ' * . -. 

(ADSS)^ which are 'equal unit measures- The analysis 'is conducted using 

-ft * - * ' , * 

AD.SS scores and then mean ADSS scores for groups are converted to raw 
scores and then to percentiles* In this way, more meaningful percentiles 
ate reported and the use o£ grade equivalent scores for evaluation 
purpose^, is eliminated* 

At present, the rest of the information available (financial, project 



a * 

variables, parental involvement information, ^etc*,) is treatfed descrip-* 
tively or in simple crosstabulation form. Future directions "Vill b'ia to 
relate relevant variables collected to the achievment gains demonstrated. 

JTXTURg DIRECTIONS * . / . . ^- " ^ i 

Since; the. system is relatively young it has been impossible to, , * 

■ * f - ' ' ' ^ 

design ceiftain highly important, elements intb^it* It^is reasonably clear 
that an efficient, responsive and comprehensive system , *has been 

developed and that ,a series of next steps can'be attempted* y 

^ ■ ' ' ' ■ ■ ■ > - ^ 

Future "eSforts will InclMe the fbllQwingi'^ * " - 

1/ Using the variables available in the system, we can begin to 

> 

deliheate successful as well as unsuccessful elements of compen- o 
satory education readitig programs. 

2* Analysis in future years will^include the area of mathematics 
achievement as a basic skill* ' " *■ 

3. An attempt will be made to incorporate one or more of the evaluation 
models for Title I recently developed by KMC Research Corporation in 
an effort to strengthen statements laade about the cognitive progress 
of students iUN the programs* ^ . 



4* An effort T/ill be made to delineate the numberj nature and types of 
activities that. impinge on each child* It^is possible that effects 
on achievement will be different depending on number and types of 
. ' supplemental activities the student is .engaged in- 
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5y Information In the fiFture will be gathered on the duties, functions, 
etc, of Individual building pairent advisory groups and not only on 
, district-wide PAC groups* ^ 

■ 4 * 

'J- * ■ 

It is hoped that with further development of this data collection 
^;and evaluation system we. ^13P bje" better 'abl-e to answer questionsu^iike, 
"iThat works In ^compensatory education programs?" 



' ^TABLEi^ 

' ^ Perqentile gains of kindergarten students enrolled in Title I reading or 
readlng-j;elate^ -programs » .4 - ^ ' . ' r' ' ^ ' . ' . - 

Pretest . . ^Posttest 

Percentiles > Percentiles 



' ***** ^ ' 

14<:r» - ^ - - - - - --^ - - - - -V^ - lz.-^^14 

=::rz=z:zzzrzzzzzz:**^ormal" performance of average students/* ^ 

^actual performance of-Title I participants 

J ^ _ _ ^ -expected performance without compensatory program 



TABLE 2 



Percentile g^ins o!E grade one students enrolled. in Title I rSading^; 
program* . ^ - . , , . ^ ^ " ^ ^ 



E^etest 

Percentiles " * ^ 



Posttest ; 
^Percentiles 

■» r 



r . . <44 ■ 

J 14y^^ - - - - - - - - - - - - - - - - 14 

^ ft 

:z"normar' performance of average stirdentcs . r 

_j ^actual performance of Tffci^^ I pa-rticiparfts ^. 

- - -^^xpecteU performance TArithout. compensatory program 
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: ^ . ' - TABLE 3 

* % 

Percentile?; ga'ins of ^grade two students enrolled in Title I reading 
protrram. - ^ ' ^ 

Pretest . ' * . ^ ^ . Posttest 

' Pe*x;entil^s ' ' ^ , Percentiles 

t " ' 

20^=" - - - _ - - - - - - - - - - - - - .2 

^ . ^ ' , ■ * * 

'.=!=i=:=============t=s=:"normar' performar^ce of average stpdents, 

[ ^ _actual performance o£%itle^I participants ^ 

■- - - - - - -expected performance wjtbput corppensatory program ' 
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TABLE 4 



Percentile gains in vocabulary, comprehension and total readijrig for grades 
two to eight. 



Grande 


Vocabulary 


Compreliension 


Total Reading\^ 
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